3D SEM Surface Reconstruction: An Optimized, Adaptive, and Intelligent Approach

Structural analysis of microscopic objects is a longstanding topic in several scientific disciplines, including biological, mechanical, and material sciences. The scanning electron microscope (SEM), as a promising imaging equipment has been around to determine the surface properties (e.g., compositions or geometries) of specimens by achieving increased magnification, contrast, and resolution greater than one nanometer. Whereas SEM micrographs still remain two-dimensional (2D), many research and educational questions truly require knowledge and information about their three-dimensional (3D) surface structures. Having 3D surfaces from SEM images would provide true anatomic shapes of micro samples which would allow for quantitative measurements and informative visualization of the systems being investigated. In this research project, we novel design and develop an optimized, adaptive, and intelligent multi-view approach named 3DSEM++ for 3D surface reconstruction of SEM images, making a 3D SEM dataset publicly and freely available to the research community. The work is expected to stimulate more interest and draw attention from the computer vision and multimedia communities to the fast-growing SEM application area

AI-Powered Analysis of Adverse Drug Events using Big Data Biomedical Literature and Deep Learning Neural Networks

Adverse drug events (ADEs) as a set of detriments caused by medications, have led to additional medical cost, prolonged hospitalization, disability, and morbidity worldwide. The study of ADEs has been a longstanding topic in the medical literature, where several medical data sources, such as EHRs, medical case reports, and drug reviews have been widely employed to interpret the ADEs. In recent years, a vast number of scientific articles and health-related social media posts get published on a daily basis, however little is known about the contents associated with perusing adverse reactions resulting from medications. Of late, artificial neural networks have been increasingly used to discover hidden patterns and facts from massive and diverse clinical data sources, and they have demonstrated promising results in a variety of applications ranging from named entity recognition and clinical note understanding to computational phenotyping. In this presentation, we will discuss, from the computational side, the state-of-the-art big data deep neural network components we developed in the context of analyzing ADEs using biomedical literature. This presentation will consist of our recent findings and implementations, ongoing research and the future directions in using deep artificial neural networks to ADEs analysis

Three-dimensional reconstruction of highly complex microscopic samples using scanning electron microscopy and optical flow estimation - Fig 6

<p>Comparison of the results for <i>Graphene</i>: a) the overall as well as a zoomed region of the computed disparity map using the state-of-the-art method of [<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0175078#pone.0175078.ref032" target="_blank">32</a>] which uses sparse feature-based matching approach and <i>a contrario</i> RANSAC for outlier removal. The dense disparity map is created by scattered data interpolation of the sparse disparity values. b) the result of Horn/Schunck optical flow estimation [<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0175078#pone.0175078.ref043" target="_blank">43</a>], which provides a better estimation of the disparity map than that of [<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0175078#pone.0175078.ref032" target="_blank">32</a>]. c) the result of dense feature matching proposed in [<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0175078#pone.0175078.ref068" target="_blank">68</a>] which uses dense SIFT features as well as factor graph representation of the matching energy functional optimized by loopy belief propagation. Even though relatively better than [<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0175078#pone.0175078.ref043" target="_blank">43</a>], the result still suffers from blurred edges. The result of the proposed method is presented in (d). In comparison to the state-of-the-art, the proposed approach generates a sharper and more accurate disparity map.</p

Effects of weighted median filtering on the horizontal disparity map: a) before and b) after.

<p>Despite inclusion of non-local term in the optical flow energy functional, the outcome can be improved greatly by adding an additional weighted median filtering step.</p

Three-dimensional reconstruction of highly complex microscopic samples using scanning electron microscopy and optical flow estimation - Fig 7

<p>Comparison of the results for <i>Fly Ash</i>: a) the overall as well as a zoomed region of the computed disparity map using the state-of-the-art method of [<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0175078#pone.0175078.ref032" target="_blank">32</a>] which uses sparse feature-based matching approach and <i>a contrario</i> RANSAC for outlier removal. The dense disparity map is created by scattered data interpolation of the sparse disparity values. b) the result of Horn/Schunck optical flow estimation [<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0175078#pone.0175078.ref043" target="_blank">43</a>], which provides a better estimation of the disparity map than that of [<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0175078#pone.0175078.ref032" target="_blank">32</a>]. c) the result of dense feature matching proposed in [<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0175078#pone.0175078.ref068" target="_blank">68</a>] which uses dense SIFT features as well as factor graph representation of the matching energy functional optimized by loopy belief propagation. Even though relatively better than [<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0175078#pone.0175078.ref043" target="_blank">43</a>], the result still suffers from blurred edges. The result of the proposed method is presented in (d). In comparison to the state-of-the-art, the proposed approach generates a sharper and more accurate disparity map.</p

Relationship between the estimated height (<i>h</i>) and the computed horizontal disparity (<i>d</i>) using the pixel size in sample units (<i>p</i>) and the total tilt angle (<i>θ</i>).

<p>Relationship between the estimated height (<i>h</i>) and the computed horizontal disparity (<i>d</i>) using the pixel size in sample units (<i>p</i>) and the total tilt angle (<i>θ</i>).</p

Dataset acquired using a Hitachi S-4800 Field Emission Scanning Electron Microscope (FE-SEM) by tilting the specimen stage by 7°.

<p>The samples are (a) <i>Arabidopsis Anther 1</i> (1280 × 960), (b) <i>Arabidopsis Anther 2</i> (1280 × 960), (c) <i>Graphene</i> (1280 × 960), (d) <i>Pseudoscorpion</i> (960 × 1280) and (e) <i>Fly Ash</i> (926 × 924). The micrographs for the <i>Pseudoscorpion</i> set are rotated by 90° for visualization purposes.</p

Rectification results: number of SIFT points found in each input image (rows 1 and 2), number of matching points after <i>a contrario</i> RANSAC (row 3), initial and final rectification errors from before and after the quasi-Euclidean rectification (rows 4 and 5).

<p>As can be seen, despite careful image acquisition, the initial rectification errors are large.</p

From start to finish: (a) first image from the <i>Fly Ash</i> sample set, (b) 3D solid model generated using the computed disparity estimates and modified using MeshMixer [70], (c) 3D printed model.

<p>Using the proposed approach, highly complex structure of the sample was captured and reconstructed in the printed model.</p

Optical flow estimation results for (a) <i>Arabidopsis Anther 1</i>, (b) <i>Arabidopsis Anther 2</i>, (c) <i>Graphene</i>, (d) <i>Pseudoscorpion</i> and (e) <i>Fly Ash</i> sample sets.

<p>The first row shows the initial difference maps. The second row shows the computed optical flow estimate. Using the optical flow estimate, the first image in each pair is warped and then used for generating the final difference maps as depicted in the third row. It should be noted that the images for <i>Pseudoscorpion</i> set are rotated by 90° for visualization purposes.</p